Systems, apparatus, methods and computer-readable storage media facilitating surgical procedures utilizing augmented reality

ABSTRACT

Techniques facilitating augmented reality-assisted surgery are provided. In one example, a method is provided that includes receiving, by a first device including a processor, image data associated with an external portion of a tool located within a body of a patient, wherein the image data includes first information indicative of a first fiducial marker on the external portion of the tool. The method also includes determining one or more relative positions of an internal portion of the tool within the body relative to one or more anatomical structures of the body based on the image data and a defined configuration of the tool. The method also includes generating one or more representations of the tool within the body relative to the one or more anatomical structures based on the one or more relative positions and the defined configuration of the tool.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Division of U.S. patent application Ser. No.15/236,750, filed Aug. 15, 2016, entitled “ SYSTEMS, APPARATUS, METHODSAND COMPUTER-READABLE STORAGE MEDIA FACILITATING SURGICAL PROCEDURESUTILIZING AUGMENTED REALITY,” which claims the benefit of the filingdate of U.S. Provisional Application No. 62/210,806, filed Aug. 27,2015, and entitled, “SYSTEMS, APPARATUS, METHODS AND COMPUTER-READABLESTORAGE MEDIA FACILITATING SURGICAL PROCEDURES UTILIZING AUGMENTEDREALITY,” the content of both of which are incorporated herein byreference in their entirety.

TECHNICAL FIELD

This disclosure relates generally to medical devices and, moreparticularly, to systems, apparatus, methods and computer-readablestorage media facilitating surgical procedures utilizing augmentedreality.

BACKGROUND

Contemporary healthcare relies heavily on implantable medical devices(IMDs) to assist patients in leading healthy lives. Implantation of suchIMDs as well as exploratory procedures for diagnosis and/or treatment ofmedical ailments typically involve surgical procedures contactinginternal areas of a body of a patient. If these procedures are performedincorrectly or with the improper tools, damage can occur to the heart,lungs and other internal organs of the body. Further, these internalareas can be difficult to see because anatomical structures may obstructthe view of the surgeon. However, making large incisions in the body ofa patient to facilitate the view of the surgeon is often undesirable.Accordingly, there is a desire for approaches that facilitate surgicalprocedures utilizing augmented reality.

SUMMARY

The following presents a simplified summary of one or more of theembodiments in order to provide a basic understanding of one or more ofthe embodiments. This summary is not an extensive overview of theembodiments described herein. It is intended to neither identify key orcritical elements of the embodiments nor delineate any scope ofembodiments or the claims. Its sole purpose is to present some conceptsof the embodiments in a simplified form as a prelude to the moredetailed description that is presented later. It will also beappreciated that the detailed description may include additional oralternative embodiments beyond those described in the Summary section.

Embodiments described herein include systems, apparatus, methods andcomputer-readable storage media facilitating surgery using augmentedreality. The term “surgery” as used herein refers to the treatment orevaluation of injuries, medical conditions or disorders of the body byincision or manipulation, especially with instruments. In someembodiments, the surgery is a substernal lead implantation procedure.However, many of the disclosed embodiments can be applied to othersurgical procedures involving an instrument.

In one embodiment, a method is provided. The method can includereceiving, by a first device including a processor, image dataassociated with an external portion of a tool located within a body of apatient, wherein the image data comprises first information indicativeof a first fiducial marker on the external portion of the tool; anddetermining one or more relative positions of an internal portion of thetool within the body relative to one or more anatomical structures ofthe body based on the image data and a defined configuration of thetool. The method can also include generating one or more representationsof the tool within the body relative to the one or more anatomicalstructures based on the one or more relative positions and the definedconfiguration of the tool.

In some embodiments, the image data also includes second informationindicative of an external part of the body, and the determining the oneor more relative positions of the internal portion of the tool caninclude determining one or more positions of the first fiducial markerrelative to the external part of the body. The image data can alsoinclude second information indicative of a second fiducial markerlocated on an external part of the body, and the determining the one ormore relative positions of the internal portion of the tool can includedetermining one or more positions of the first fiducial marker relativeto the second fiducial marker.

In some embodiments, the patient is a human and the second fiducialmarker is located on an external area of the body over and adjacent to asternum of the patient. By way of example, but not limitation, the toolcan be inserted into the body via an incision in the body and the secondfiducial marker can be located proximate to the incision.

In some embodiments, the tool includes an elongated shaft having aplurality of markers and the image data includes one or more visiblesubsets of the plurality of markers. Determining the one or morerelative positions of the internal portion of the tool within the bodycan be based on determining the one or more visible subsets of theplurality of markers relative to the plurality of markers. In someembodiments, determining the one or more relative positions of theinternal portion of the tool within the body includes determining anorientation of the internal portion of the tool within the body.

In some embodiments, the method also includes displaying, by the firstdevice, the one or more representations on a first display of the firstdevice, wherein the one or more representations facilitate guidance ofthe tool to a target location within the body. In some embodiments, asecond device performs the display of the one or more representations.

In another embodiment, a medical device is provided. The medical devicecan include an elongated shaft comprising a distal end and a proximalend, wherein the distal end is configured for insertion into a body of apatient; a handle coupled to the proximal end of the elongated shaft;one or more fiducial markers located on the elongated shaft; and anorientation unit including an accelerometer configured to collectorientation information regarding an orientation of the medical deviceand provide the orientation information to another device, wherein anorientation and position of the medical device relative to the body ofthe patient are determined based on the orientation information andimage data captured by one of the one or more fiducial markers.

The one or more fiducial markers can be configured to facilitate opticaltracking of one or more positions of an internal portion of theelongated shaft inside the body of the patient. In some embodiments, oneor more fiducial markers are located on an outer surface of theelongated shaft. In some embodiments, one or more fiducial markers arepositioned to be detected by a single stationary camera irrespective ofan orientation of the medical device relative to the body of thepatient.

In yet another embodiment, a tangible computer-readable storage mediumis provided. The computer-readable storage medium can includecomputer-readable instructions that, in response to execution, cause adevice to perform operations. The operations can include: receivingimage data associated with an external portion of a medical deviceinserted within a body of a patient, wherein the image data comprisesfirst information associated with a first fiducial marker on theexternal portion of the medical device; and determining a relativeposition of an internal portion of the medical device within the bodyrelative to one or more anatomical structures of the body based on theimage data and a configuration of the medical device. In someembodiments, the operations can include generating a representation ofthe internal portion of the medical device within the body relative tothe one or more anatomical structures based on the relative position andthe configuration of the medical device.

Other embodiments and various non-limiting examples, scenarios andimplementations are described in more detail below. The followingdescription and the drawings set forth certain illustrative embodimentsof the specification. These embodiments are indicative, however, of buta few of the various ways in which the principles of the specificationmay be employed. Other advantages and novel features of the embodimentsdescribed will become apparent from the following detailed descriptionof the specification when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of an example, non-limitingsystem facilitating surgery using augmented reality in accordance withone or more embodiments described herein.

FIG. 2 illustrates a schematic diagram of an example, non-limitinginstrument that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein.

FIG. 3 illustrates a schematic diagram of an example, non-limitinginstrument that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein.

FIG. 4 illustrates a schematic diagram of an example, non-limitinginstrument that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein.

FIG. 5 illustrates a schematic diagram of another example, non-limitingsystem that facilitates surgery using augmented reality in accordancewith one or more embodiments described herein.

FIG. 6 illustrates a schematic diagram of another example, non-limitingsystem that facilitates surgery using augmented reality in accordancewith one or more embodiments described herein.

FIG. 7 illustrates a schematic diagram of an example extravascular ICDsystem implanted within a body of a patient in accordance with one ormore embodiments described herein.

FIG. 8 illustrates a schematic diagram of an example, non-limitinginstrument that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein.

FIG. 9 illustrates a schematic diagram of another example, non-limitingsystem that facilitates surgery using augmented reality in accordancewith one or more embodiments described herein.

FIG. 10 illustrates a schematic diagram of another example, non-limitingsystem that facilitates surgery using augmented reality in accordancewith one or more embodiments described herein.

FIG. 11 illustrates a schematic diagram of another example, non-limitingsystem that facilitates surgery using augmented reality in accordancewith one or more embodiments described herein.

FIG. 12 illustrates a block diagram of an example, non-limiting imagingdevice facilitating surgery using augmented reality in accordance withone or more embodiments described herein.

FIG. 13 illustrates a schematic diagram of an example, non-limitingvisualization that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein.

FIGS. 14A and 14B illustrate schematic diagrams of example, non-limitinginstruments that facilitate surgery using augmented reality inaccordance with one or more embodiments described herein.

FIG. 15 illustrates a flow diagram of an example, non-limiting methodfacilitating surgery using augmented reality in accordance with one ormore embodiments described herein.

FIG. 16 illustrates a flow diagram of another example, non-limitingmethod facilitating surgery using augmented reality in accordance withone or more embodiments described herein.

FIG. 17 illustrates a flow diagram of another example, non-limitingmethod facilitating surgery using augmented reality in accordance withone or more embodiments described herein.

FIG. 18 illustrates a block diagram of an example, non-limitingenvironment including a computer operable to facilitate surgery usingaugmented reality in accordance with one or more embodiments describedherein.

DETAILED DESCRIPTION

The following detailed description is merely illustrative and is notintended to limit embodiments and/or application or uses of embodiments.Furthermore, there is no intention to be bound by any expressed orimplied information presented in the preceding Technical Field,Background or Summary sections, or in the Detailed Description section.

One or more embodiments are now described with reference to thedrawings, wherein like referenced numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea more thorough understanding of the one or more embodiments. It isevident, however, in various cases, that the one or more embodiments canbe practiced without these specific details.

Additionally, the following description refers to components being“connected” and/or “coupled” to one another. As used herein, unlessexpressly stated otherwise, the terms “connected” and/or “coupled” meanthat one component is directly or indirectly connected to anothercomponent, mechanically, electrically, wirelessly, inductively orotherwise. Thus, although the figures may depict example arrangements ofcomponents, additional and/or intervening components may be present inone or more embodiments.

With reference now to the drawings, FIG. 1 illustrates a schematicdiagram of an exemplary, non-limiting system facilitating surgery usingaugmented reality in accordance with one or more embodiments describedherein. Aspects of systems, apparatuses and/or processes explained inthis disclosure can include machine-executable components embodiedwithin machine(s), e.g., embodied in one or more computer readablemediums (or media) associated with one or more machines. Suchcomponents, when executed by the one or more machines, e.g.,computer(s), computing device(s), virtual machine(s), etc., can causethe machine(s) to perform the operations described.

In the embodiment shown, system 100 includes a patient 102, a surgicalinstrument 106, an image capture device 114 and an imaging device 116.In various embodiments, one or more of the surgical instrument 106,image capture device 114 and/or imaging device 116 can be electricallyand/or communicatively coupled to one another to perform one or morefunctions of system 100.

System 100 incorporates augmented reality technology with surgicalapparatus and/or procedures (involving instrument 106) to provide visualimage guidance in association with employing and/or navigating theinstrument 106 inside a body 102 of the patient. In an embodiment,system 100 can enable real-time tracking and/or visualization of theportion of the instrument 106 that is obstructed from external view ofthe operator of the instrument 106. For example, a portion of theinstrument 106 can be obstructed from view due to insertion into an areaof the body 102 that is not visible to the operator of the instrument106 (e.g., surgeon or otherwise) or that is partially occluded bytissue, body fluids or the like as the instrument 106 is being used toperform surgery. In particular, the position and/or orientation of theinstrument 106 inside the body 102 relative to various internal and/orexternal anatomical structures of the body 102 can be determined as theinstrument 106 is being maneuvered inside the patient (e.g., in-vivo).In some embodiments, the position and/or orientation of the instrument106 can be determined in real-time (e.g., within a few seconds ormilliseconds after positioning the instrument or after a change inpositioning of the instrument 106).

Electronic information indicative of two-dimensional (2-D) and/or threedimensional (3-D) representations of the instrument 106 inside the body102 can be generated and/or displayed. In some embodiments, therepresentations can be integrated within a display showing a 2-D or 3-Dvisual model of the body 102 and the internal and/or external anatomicalstructures of the body 102.

For example, a 2-D and/or 3-D model of internal and/or externalanatomical structures of the patient can be generated (by system 100 oranother system) based on previously captured image data of the patientand/or known anatomical structures of the human body. The previouslycaptured image data of the body 102 of the patient can include imagedata captured using various medical imaging technologies such as but notlimited to: X-ray radiography, magnetic resonance imaging, medicalultrasonography or ultrasound, endoscopy, elastography, tactile imaging,thermography, medical photography (e.g., X-ray computed tomography(X-ray CT) or computerized axial tomography scan (CAT scan)) and/ornuclear medicine functional imaging techniques such as positron emissiontomography.

In various embodiments, the visual model generated by system 100 canvary in complexity based on the area of the body 102 in which theinstrument 106 is being used and/or the type of surgical operation beingperformed. For example, the visual model can include various internaland/or external anatomical structures of the body 102. The internaland/or external anatomical structures can include, but are not limitedto, bones, muscles, organs, tissues, glands, ligaments, tendons, nerves,veins, and vessels. Foreign structures located inside the body 102, suchas IMDs, can also be included in the model in some embodiments.

In accordance with various embodiments of system 100, as an instrument(e.g., instrument 106) is located within, inserted into and/ormaneuvered within the body 102 of the patient, 2-D and/or 3-Drepresentations of the in-vivo portion of the instrument 106 can begenerated. In some embodiments, the 2-D and/or 3-D representations ofthe in-vivo portion of the instrument 106 can be integrated within avisual model of the body 102 at the actual position of the instrument106 and/or orientation relative to the modeled anatomical structures.This visualization of the instrument 106 as the instrument 106 appearsinside the body 102 can be displayed for view by an operator of theinstrument 106 (e.g., surgeon) via a display device, as a holographicprojection or any other approaches for displaying information.Accordingly, the surgeon can view the instrument 106 inside the body 102relative to anatomical landmarks without the need to expose theinstrument 106 and/or anatomical landmarks (e.g., via further cutting oropening of the body 102). In some embodiments, the displayed informationcan be provided in real-time (e.g., within 1-3 seconds after theinformation is gathered in-vivo). As used in this disclosure, the term“user” can refer to a person, entity, device, or system, or acombination of a person, entity, device, or system. For example, in someembodiments, a user can include a robot having a robotic arm fullycontrolled or controlled in part by a computing device or by a surgeonto hold and/or maneuver the instrument 106 during surgery. In someembodiments, the user is a surgeon.

In accordance with various embodiments, system 100 is configured toemploy optical tracking to enable visualization of a portion ofinstrument 106 that is in-vivo relative to one or more internal and/orexternal anatomical structures of the body 102. According to theseembodiments, an image capture device 114 is positioned outside the body102 of the patient and captures image data of the portion of theinstrument 106 that is located ex-vivo when the instrument 106 islocated within, inserted into and/or maneuvered within the body 102 ofthe patient.

The image data can also include information indicative of one or moreother external features and/or structures included in the operatingenvironment, such as an external portion of the body 102 of the patient,a portion of the operating table upon which the body 102 of the patientis rested, a fiducial marker that has been placed on the body 102 of thepatient (e.g., sternal fiducial marker 104 and/or incision fiducialmarker 120) or the like. As used herein, the term “external” meansoutside of the body, or ex-vivo. One or more of the external featuresand/or structures can serve as one or more reference points tofacilitate determining a spatial relationship between the instrument106, the reference points, actual anatomical structure of the body 102,and/or visually modeled structures of the body 102, as the instrument106 is located inside the body 102.

In some embodiments, the captured image data can be provided to animaging device 116. For example, the captured image data can betransmitted via wireless or wired link from the image capture device 114to the imaging device 116. The imaging device 116 can be configured toprocess the image data and determine a position and/or orientation ofone or more portions of the instrument 106. For example, in someembodiments, the imaging device 116 can be configured to process theimage data and determine a position and/or orientation of the in-vivoportion of the instrument 106 relative to the one or more internaland/or external anatomical structures of the body 102.

In some embodiments, the imaging device 116 can generate 2-D and/or 3-Drepresentations of the in-vivo portion (and/or the ex-vivo portion) ofthe instrument 106. In some embodiments, the imaging device 116 canintegrate the 2-D and/or 3-D representations of the instrument 106within a previously generated 2-D and/or 3-D model of the internaland/or external structures of the body 102 at the determined positionand/or orientation of the portion of the instrument 106 relative to themodeled anatomical structures.

In some embodiments, the imaging device 116 can determine the positionand/or orientation of the in-vivo portion of the instrument 106 relativeto the one or more anatomical structures based on the captured imagedata associated with the ex-vivo portion of the instrument 106, as wellas a known configuration (e.g., size and/or shape) of the instrument106, a known position of a captured reference point relative to a 2-D or3-D coordinate space, and/or known positions of the one or moreanatomical structures relative to the 2-D or 3-D coordinate space. Inone example, the imaging device 116 can compare the size and/or shape(as determined via the image data) of the ex-vivo portion of theinstrument 106 to a known model (size and shape) of the entireinstrument 106. The portion of the known model of the instrument 106that is not included in the captured ex-vivo portion can be determinedby the imaging device 116 to correspond to the in-vivo portion of theinstrument 106. The imaging device 116 can then generate arepresentation of the in-vivo portion and combine the representation ofthe in-vivo portion with the real image data or a model of the ex-vivoportion.

In some embodiments, the imaging device 116 can determine the positionand/or orientation of the in-vivo portion of the instrument 106 relativeto the one or more anatomical structures of the body 102 based on adetermined position and/or orientation of the ex-vivo portion of theinstrument 106 relative to a defined reference point or object.According to this embodiment, the actual positions and orientations ofthe one or more internal and/or external anatomical structures of thepatient relative to a 2-D or 3-D coordinate space can be determined. Forexample, the 2-D or 3-D model of the internal and/or external anatomicalstructures of the body 102 can be aligned or correlated to an actualposition and/or orientation of the body 102 of the patient relative to a2-D or 3-D coordinate space. A relative position of an actual referencepoint or object in the 2-D or 3-D coordinate space can also be known ordefined. The imaging device 116 can thus determine a position and/ororientation of the ex-vivo portion of the instrument 106 in thecoordinate space relative to the one or more reference points and employtriangulation to determine the position and/or orientation of theex-vivo portion of the instrument 106 relative to the modeled anatomicalstructures.

In some implementations, the imaging device 116 can determine theposition and/or orientation of the in-vivo portion of the instrument 106relative to the modeled anatomical structures based on the positionand/or orientation of the ex-vivo portion relative to the modeledanatomical structures and a known or defined geometry of the instrument106.

In some embodiments, the captured image data can include depthinformation for the instrument 106. The depth information can correspondto a distance between the image capture device 114 and features (e.g.,pixels) included in the captured image data (e.g., points along the exvivo portion of the tool, points on the reference object, points on thebody 102 of the patient, etc.). The imaging device 116 can employ thisdepth data to determine the relative positions and/or orientations ofthe ex-vivo portion of the instrument 106 relative to a 3-D coordinatespace and a reference point included in the captured image data forwhich the position relative to the 3-D coordinate space is known. Theimaging device 116 can then determine the position and/or orientation ofthe in-vivo portion of the instrument 106 relative to the 3-D coordinatespace based on the determined position and/or orientation of the ex-vivoportion relative to the 3-D coordinate space.

In addition to or as an alternative to the optical tracking and/orvisualization/display techniques discussed herein, in some embodiments,system 100 can perform optical tracking of the ex-vivo portion of theinstrument 106 using visible fiducial markers on the instrument 106. Forexample, in various exemplary embodiments, the instrument 106 caninclude one or more fiducial markers. In some embodiments, the one ormore fiducial markers (not shown) can include a pattern that is definedand/or known by the imaging device 116.

At least one fiducial marker can be provided at a location on theinstrument 106 adapted or intended to remain outside of the body 102when the instrument 106 is inserted inside the body 102. As such, thefiducial marker is capable of being captured by the image capture device114. For example, the instrument 106 can include a handle 110 and aplatform 108 near the handle 110 that is configured to remain outside ofthe body 102 of the patient during use of the instrument 106. Accordingto this example, a fiducial marker can be located on the platform 108.Fiducial marker 200 will be shown and described with reference to FIG.2, for example.

With reference to FIG. 1, according to one or more of these embodiments,the image capture device 114 can be configured to capture image data ofthe ex-vivo portion of the instrument 106, including the fiducialmarker(s) on the ex-vivo portion of the instrument 106. The imagecapture device 114 can be also configured to capture image data of atleast one reference point or object near or within a defined distance ofthe instrument 106 (e.g., within the visual capture space of the imagecapture device 114) having a known position relative to a 2-D or 3-Dcoordinate space. In some embodiments discussed, the reference point orobject can include a second fiducial marker placed on the body 102 ofthe patient (e.g., sternal fiducial marker 104 and/or incision fiducialmarker 120). With reference to FIG. 1, the imaging device 116 canreceive and/or analyze the image data to determine a position and/ororientation of the instrument 106 relative to a modeled representationof the internal and/or external anatomical structures of the body 102 ofthe patient that has been correlated to the 2-D or 3-D coordinate space.In various embodiments, the imaging device 116 can receive and/oranalyze the image data to determine a position and/or orientation of theinstrument 106 relative to a modeled representation of the internaland/or external anatomical structures of the body 102 of the patientthat has been correlated to the 2-D or 3-D coordinate spacenotwithstanding a portion of the instrument 106 is located in-vivo.

In some embodiments, the imaging device 116 determines the size and/orshape of the entire instrument 106, or receives previously determinedinformation that defines the size and/or shape of the entire instrument106 or one or more substantial portions of the instrument 106. Forexample, in various embodiments, the imaging device 116 can determinethe size and/or shape of 75%, 95% or 100% of the entirety of theinstruments.

In some embodiments, the imaging device 116 can determine or receivepreviously determined data that defines, or that can be employed todefine, the three-dimensional geometry of the instrument 106. Dataregarding the structure and/or geometry of the instrument 106 can alsoidentify a position of at least one fiducial marker of the instrument106 relative the structure and/or geometry of the instrument 106.According to this embodiment, the imaging device 116 can be configuredto recognize the unique pattern of the fiducial marker in the capturedimage data and determine a position and orientation of the fiducialmarker in the 2-D or 3-D coordinate space by mapping the position andorientation of the fiducial marker to the known position of one or morereference points in the 2-D or 3-D coordinate space. Using the knownrelationship between the fiducial marker and the 2-D and/or 3-D geometryof the instrument 106, the imaging device 116 can further determine theposition and orientation of the in-vivo and ex-vivo portions of theinstrument 106 relative to the 2-D and/or 3-D coordinate space. In someembodiments, the imaging device 116 can employ depth informationincluded in the image data that identifies a distance from the imagecapture device to the fiducial marker and/or the reference point tofacilitate calculating the current position and/or orientation of thefiducial marker.

In some implementations, when the image capture device 114 is providedat a fixed position throughout the surgical procedure, a vantage pointof the image capture device relative to an appearance of the fiducialmarker in the captured image data can also be used to determine anorientation of the instrument 106. For example, the imaging device 116can determine or be provided with information that correlates variousrotational positions of the instrument 106 in a 3-D coordinate spacewith specific appearances of the fiducial marker and/or a visible anddistinguishing portion of a pattern of the fiducial marker.

Once the position and/or orientation of the instrument 106 (or theex-vivo portion of the instrument 106) relative to the 2-D and/or 3-Dcoordinates space is determined, the position and/or orientation of thein-vivo portion of the tool can be mapped to a model of the anatomicalstructures of the patient when the model is correlated to the same 2-Dand/or 3-D space using the techniques discussed above. For example, theimaging device 116 can determine an actual position and/or orientationof the body 102 of the patient relative to the 2-D and/or 3-D coordinatespace. Imaging device 116 can further superimpose a known model of theanatomical features of the body 102 of the patient on the same 2-Dand/or 3-D coordinate space.

In some embodiments, after the location and/or orientation of theex-vivo portion of the instrument 106 relative to the 2-D and/or 3-Dcoordinate space is determined, the position and/or orientation of thein-vivo portion relative to the 2-D and/or 3-D coordinate space isdetermined based on a known configuration (e.g., size and/or shape) ofthe instrument 106. The imaging device 116 can integrate a 2-D and/or3-D representation of the instrument 106 into the 2-D and/or 3-Danatomical visual model at the determined position and/or orientation ofthe instrument 106.

FIG. 2 illustrates a schematic diagram of an example, non-limitinginstrument 106 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. With referenceto FIGS. 1 and 2, in another embodiment, the instrument 106 can alsoinclude a plurality of markers (shown as markers 202, 204, 206 of FIG.2) along one or more portions of an elongated shaft 112 of theinstrument 106. For example, instrument 106 can include an elongatedshaft 112 that is configured to be inserted into the body 102 at variousdepths. The elongated shaft 112 can have a plurality of known markers202, 204, 206 or gradations on the length of the elongated shaft 112(e.g., akin to gradations of a ruler).

The relative positions of the markers with respect to a distal end ofthe instrument 106 (e.g., the end of the instrument 106 to be insertedinto the body 102) and a proximal end of the instrument 106 (e.g., theend of the instrument 106 having the handle 110) can be known in someembodiments. According to these embodiments, the image capture device114 can capture image data of the ex-vivo portion of the instrument 106.The imaging device 116 can analyze the image data to identify a subsetof the plurality of markers 202, 204, 206 that are present and/ordetectable in the image data. In some embodiments, using a subtractiontechnique, the image capture device 114 can determine another subset ofthe plurality of the markers 202, 204, 206 that are located in-vivo.Using a known relationship between the configuration of the instrument106 and the locations of the plurality of markers 202, 204, 206 on theinstrument 106 relative to the distal and proximal ends of theinstrument 106, the imaging device 116 can determine the portion orlength of the instrument 106 that is located in-vivo.

In various embodiments, as described herein, instrument 106 can be,correspond to or be coupled to a tunneling tool employed to generate aroute for implantation of one or more IMDs. In one embodiment, the IMDcan be or include a lead of an extravascular implantable cardioverterdefibrillator (ICD) system having electrodes placed underneath/below thesternum/ribcage.

With reference to FIG. 2, as shown, the elongated shaft 112 can have acurved geometry in some embodiments. In other embodiments, any number ofother suitable geometries of the elongated shaft 112 can be employed.

In some embodiments, the geometry and/or length of the elongated shaft112 can be adjusted by a user of the instrument 106 prior toimplantation into the body 102 to suit a particular anatomy of thepatient. For example, the shaft 112 can be configured to be shaped intoa plurality of different configurations having predefined geometries.According to this example, the imaging device 116 can detect anddetermine the particular configuration and associated geometry of theinstrument 106 prior to insertion based on image data captured of theinstrument 106 by image capture device 114, and/or a user of theinstrument 106 can provide input to the imaging device 116 identifyingthe particular configuration of the instrument 106. The elongated shaft112 can include a plurality of markers 202, 204, 206. While threemarkers 202, 204, 206 are indicated, in various embodiments, the markerson the elongated shaft 112 can be any number and are generally indicatedby striped pattern on the elongated shaft 112 shown in FIG. 2.

In some embodiments, the distances between the respective markers andthe distal and proximal ends of the instrument 106 and/or the distancesbetween each of the markers can be predetermined and/or known. Thesemarkers (e.g., markers 202, 204, 206) can facilitate determining thelength of a portion of the instrument 106 that is located in-vivo duringuse. The instrument 106 further includes a fiducial marker 200 locatedon a platform 108 that extends from a portion of the elongated shaft112.

FIG. 3 illustrates a schematic diagram of another example, non-limitinginstrument 106 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. As shown inFIG. 3, in addition to the fiducial marker 200 located on platform 108,the instrument 106 includes another fiducial marker 300 on anotherplatform 302. Repetitive description of like elements employed inrespective embodiments disclosed herein is omitted for sake of brevity.

In some embodiments, as shown, platform 302 can have a differentposition and orientation relative to the elongated shaft 112 of theinstrument 106. In accordance with this embodiment, the fiducial markers200, 300 located on the respective platforms 108, 302 are different fromone another. By including two or more different fiducial markers atdifferent positions and/or orientations relative to the elongated shaft112, the position and orientation of the instrument 106 relative to a2-D or 3-D coordinate space can be more accurately determined as theinstrument 106 is moved and/or rotated.

FIG. 4 illustrates a schematic diagram of an example, non-limitinginstrument 106 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. As shown inFIG. 4, platform 108 has a cylindrical ring configuration formed arounda proximal end of the instrument 106. Repetitive description of likeelements employed in respective embodiments disclosed herein is omittedfor sake of brevity.

The platform 108 includes a fiducial marker 400 disposed on the platform108 and/or provided around the entire circumference of the outer wall ofthe ring. In accordance with this embodiment, the pattern of thefiducial marker 400 can be non-repetitive and/or non-uniform such thatdifferent radial areas of the outer wall have different patterns. Withthis embodiment, the orientation of the instrument 106 can be moreaccurately detected based on the radial portion of the fiducial marker400 on the ring that is included in a captured image. In someembodiments, although not depicted, the inner wall of the ring can alsoinclude a unique and/or non-uniform fiducial pattern around thecircumference of the inner wall to further facilitate determining aposition and/or orientation of the instrument 106 relative to a 3-Dcoordinate space.

Referring back to FIG. 1, in some embodiments, instrument 106 caninclude an orientation unit 126 that further facilitates accuratelydetermining an orientation of the instrument 106 relative to a 2-D or3-D coordinate space after the instrument 106 is inserted into and/ormaneuvered in the body 102 of the patient. The orientation unit 126 canbe configured to capture orientation data as the instrument 106 is movedand/or rotated. In some embodiments, the orientation unit 126 caninclude a processor that is configured to process the orientation datato determine an orientation of the instrument 106. In some embodiments,the orientation unit 126 can provide the orientation data to the imagingdevice 116 for processing. In some embodiments, the orientation unit 126can transmit the orientation data to the imaging device 116 via wirelessand/or wired channel.

In some implementations, the orientation unit 126 includes anaccelerometer and circuitry, hardware, software, or a combination ofhardware and software, associated with the operation of theaccelerometer. In some embodiments, the orientation unit 126 can alsoinclude a power source and/or a transmitter capable of wirelesslytransmitting captured or processed orientation data to the imagingdevice 116. In another embodiment, the imaging device 116 and theinstrument 106 can be communicatively and electrically coupled via acable (not shown).

In one or more embodiments, the imaging device 116 can employ theaccelerometer data to determine an accurate orientation of theinstrument 106 relative to a 2-D and/or 3-D coordinate space and areference point (or points) included in the actual operatingenvironment. The imaging device 116 can also combine the accelerometerorientation data with positional data for the instrument 106 determinedbased on captured image data (via the image capture device 114). Theimaging device 116 can also compare an orientation calculation for theinstrument 106 determined based on data obtained from the orientationunit 126 with another orientation calculation determined by the imagingdevice 116 based on captured image data (in accordance with the mannersdescribed above) to calibrate additional orientation determinationsbased on captured image data.

Image capture device 114 can include various types of 2-D and 3-Dimaging devices or cameras capable of capturing digital still imagesand/or video. The resolution of the digital still images and/or videocaptured by the image capture device 114 can vary in differentembodiments. In an embodiment, the image capture device 114 includes acamera capable of capturing depth data (e.g., a 3-D camera) forrespective pixels in a captured digital image. Examples of 3-D capturedevices include, but are not limited to, light detection and rangingcameras (LIDARs), hand-held laser line scanners, structured lightprojectors paired with cameras such as the Microsoft® Kinect, otherstructured light systems, stereo cameras with software for depthderivation, stereo cameras paired with pattern projection systems aswell as software for depth derivation, time-of-flight cameras, videocameras capable of structure-from-motion calculations, and/or lightfieldcameras.

In some embodiments, a plurality of image capture devices can becombined or a 3-D image capture device can be paired with a 2-D colorcamera to provide color detail for the captured 3-D information. Forexample, one or more image capture devices (e.g., image capture device114) can be located at various positions and/or orientations relative tothe operating area of the patient 102 such that the respective imagecapture devices can capture different perspectives of the body 102 ofthe patient, the instrument 106, and various other reference points orlandmarks. Image data from the respective cameras can also be combinedto determine a highly accurate position and/or orientation of theinstrument 106 relative to a 3-D coordinate space.

In one or more embodiments, the image capture device 114 is configuredto remain in a fixed position relative to the body 102 of the patient.The location of the image capture device 114 can vary depending on thearea of the body 102 of the patient upon which surgery is beingperformed. For example, the image capture device 114 can be positionedsuch that the operating area on the patient is substantially or directlybelow and/or perpendicular to the image capture device 114. In anotherexample, the image capture device 114 can be positioned on a side of thebody 102 of the patient such that the image capture device 114 issubstantially coplanar with the operating area. Still in other examples,the position of the image capture device 114 can be above and/or aroundthe operating area of the patient at various pitches relative to theoperating area. In other embodiments, the image capture device 114 canmove to different positions and/or orientations relative to the body 102of the patient during an augmented reality surgical procedure.

Still in yet another embodiment, the image capture device 114 can beworn by a user, such as a surgeon performing the surgery and employingthe instrument 106. For example, the image capture device 114 can beincluded in a head mount that is worn by the surgeon such that the fieldof view of the image capture device 114 corresponds to the field of viewof the surgeon. In another example, the image capture device 114 can belocated in a pair of goggles or glasses worn by the surgeon.

A described herein, with system 100, one or more visualizationsincluding a 2-D or 3-D representation of a tracked instrument 106in-vivo relative to one or more 2-D or 3-D internal and/or externalanatomical structures of the body 102 can be presented to a user via adevice display or as a 3-D hologram. In the embodiment depicted insystem 100, the imaging device 116 includes a display component 118 tofacilitate rendering these visualizations. The display component 118 caninclude hardware, software, or a combination of hardware and software,that facilitates generating a graphical interface that includes thepatient 2-D or 3-D representations and/or that facilitates generating a3-D hologram of the patient 2-D or 3-D representations. For example,display component 118 can include a display screen of the imaging device116. The location of the display screen can vary depending on thelocation, size, features, and functionality of the imaging device 116.For example, the imaging device 116 and the display screen of theimaging device 116 can be positioned in a suitable location in theoperating room remote from the surgeon and the body 102 of the patientand/or at a location that can be viewed by the surgeon during surgery.

In another example, the imaging device 116 can include a wearablecomputing device, such as a computing device that is located in a headmount display or a pair of glasses or goggles. According to thisexample, the display screen can be positioned near the field of view ofthe surgeon, such as on the lenses, on a portion of a lens of theglasses or goggles, or as an extension of the head mount display.

Still in yet another embodiment, the imaging device 116 can include awearable glasses computing device or a goggles-based computing devicewith a display component 118 configured to generate 3-D representationsof objects and/or integrate the 3-D representations onto the lenses ofthe glasses such that the 3-D representations are displayed spatiallyintegrated into the real world environment that is being seen be theuser through the glasses. For example, as the user looks through theglasses at the chest of the patient, the user can see an augmentedreality representation of at least a portion of the chest of the patientthat includes internal anatomical structures of the chest. In essence,when looking through such an imaging device, the user can see a 3-Dvisualization of the chest as though the external structure (e.g., skin)of the chest has been cut open and removed to reveal the internalstructures. According to this embodiment, the 3-D visualization of theanatomical structures can also include a 3-D representation of thein-vivo portion of the instrument 106 at the determined position and/ororientation of the instrument 106 relative to the anatomical structures.

FIG. 5 illustrates a schematic diagram of another example, non-limitingsystem 500 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. Repetitivedescription of like elements employed in respective embodimentsdisclosed herein is omitted for sake of brevity.

As depicted in system 500, the display component 118 can be locatedremote from the imaging device 116 in another device 502. For example,the imaging device 116 and the device 502 including the displaycomponent 118 can be remotely connected via a network and operateaccording to a server-client relationship. According to this example,the image capture device 114 is also remote from the imaging device 116.The image capture device can capture and provide image data to theimaging device 116 for processing. 2-D and/or 3-D representationsgenerated by the imaging device 116 based on the image data and/ororientation data captured via the orientation unit 126 (in accordancewith the mechanisms describe herein) is then sent to device 502 fordisplay by display component 118 (e.g., on a display screen or as aholographic projection).

FIG. 6 illustrates a schematic diagram of another example, non-limitingsystem 600 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. Repetitivedescription of like elements employed in respective embodimentsdisclosed herein is omitted for sake of brevity.

As shown in system 600, the imaging device 116, the image capture device114, and the display component 118 are respectively located on a samedevice (e.g., a wearable head mount computing device or agoggles/glasses-based wearable computing device). In another embodiment,which is not shown, the image capture device 114 and the displaycomponent 118 can be located on the same device while the imaging device116 is located on another remote device.

With reference to FIGS. 1, 5 and 6, depending on the implementation ofthe various components or devices of the systems described herein (e.g.,systems 100, 500, 600 and the like), one or more of the components ordevices can be communicatively coupled via a wireless network or viawired connections. In several exemplary embodiments, one or more of thecomponents and/or devices of the disclosed systems (e.g., orientationunit 126, image capture device 114, imaging device 116, displaycomponent 118, and device 502) can be configured to employ variouswireless communication protocols to communicate with one another. Forexample, the orientation unit 126 and the imaging device 116 cancommunicate using near field communication (NFC). In another example,the orientation unit 126, the image capture device 114, the imagingdevice, and/or device 502 can communicate using any of various types ofother wireless communication protocols. For example, other communicationprotocols can include, but are not limited to, a BLUETOOTH®technology-based protocol (e.g., BLUETOOTH® low energy (BTLE) protocol),an ultra-wideband (UWB) technology-based protocol, a radio frequency(RF) communication-based protocol, or any other proprietary ornon-proprietary communication protocols.

In various embodiments, communication can be facilitated over a personalarea network (PAN), a local area network (LAN) (e.g., a WirelessFidelity (Wi-Fi) network) that can provide for communication overgreater distances than the NFC protocol or provide other advantages(e.g., stronger encryption protocols). In some embodiments, the imagecapture device 114, the imaging device 116 and/or device 502 cancommunicate with one another and/or another device (e.g., a serverdevice or a tertiary device) over a wide area network (WAN) usingcellular or Hyper Text Transfer Protocol (HTTP)-based communicationprotocols (e.g., session initiation protocol (SIP)).

One or more embodiments of systems 100, 500 and 600 are described inconnection with facilitating implantation of a medical device within apatient, e.g., a medical device within a substernal space of thepatient, systems using augmented reality. In patients at high risk ofventricular fibrillation, the use of an implantable ICD system has beenshown to be beneficial at reducing the likelihood of sudden cardiacdeath (SCD). An ICD system can include an ICD, which is a batterypowered electrical shock device, that may include an electrical housingor an electrode (sometimes referred to as a “can electrode”) that iscoupled to one or more electrical lead wires placed on or within theheart. If an arrhythmia is sensed, the ICD can send a pulse via theelectrical lead wires to shock the heart and restore the normal rhythmof the heart.

Owing to the inherent surgical risks in attaching and replacingelectrical leads directly within or on the heart, extravascular ICDsystems have been devised to provide shocks to the heart without placingelectrical lead wires within the heart or attaching electrical wiresdirectly to the heart. Instead, the extravascular ICD systems candeliver shocks to the heart by the use of a defibrillation lead havingelectrodes placed subcutaneously between the skin and theribcage/sternum or electrodes placed substernally (e.g.,underneath/below the sternum), or a combination thereof

FIG. 7 illustrates an example extravascular ICD system 718 implantedwithin a body 102 of a patient. An extravascular ICD system such assystem 718 generally includes an ICD 704 and an electrical lead 706. TheICD 704 is provided in a housing (e.g., a titanium case) containing abattery and electronic circuitry that provides defibrillation therapyand pacing. For example, the housing can house one or more processors,memories, transmitters, receivers, sensors, sensing circuitry, therapycircuitry, power sources and other appropriate components. The ICD 704can be implanted subcutaneously within the body 102 of the patient, suchas on the left midaxillary of the patient above the ribcage (e.g., nearthe left armpit of the body 102 of the patient). The ICD 704 can also beimplanted at other subcutaneous locations on the patient. In someinstances, the ICD may alternatively be placed at a substernal location.

The electrical lead 706 includes an elongated body having a proximal endthat includes a connector that connects to the ICD 704 and a distal endthat includes a plurality of electrodes (e.g., electrodes 712, 714 and716). When properly implanted, the lead 706 extends subcutaneously abovethe ribcage from the ICD 704 toward a center of the torso of the patient(e.g., toward the xiphoid process 710 of the patient). For substernallead placements, at a location near the center of the torso, the lead706 bends or turns and extends superior under/below the sternum 702 ofthe patient such that the portion of the lead 706 containing theelectrodes 712, 714 and 716 is within the substernal space, includingthe anterior mediastinum. The anterior mediastinum can include the areawithin the body 102 being bounded laterally by the pleurae, posteriorlythe pericardium, and anteriorly by sternum 702. In some instances, theanterior wall of anterior mediastinum can also be formed by thetransversus thoracis and one or more costal cartilages.

The anterior mediastinum includes a quantity of loose connective tissue(e.g., areolar tissue), some lymph vessels, lymph glands, substernalmusculature (e.g., transverse thoracic muscle), branches of the internalthoracic artery, and the internal thoracic vein. In one example, thedistal portion of lead 706 is configured to be implanted substantiallywithin the loose connective tissue and/or substernal musculature ofanterior mediastinum. In other embodiments, the distal portion of thelead 706 can be implanted in other non-vascular, extra-pericardiallocations, including the gap, tissue, or other anatomical featuresaround the perimeter of and adjacent to, but not attached to, thepericardium or other portion of heart and not above sternum or ribcage.

Implantation of the lead 706 can be a difficult and intricate procedure,and if done incorrectly or with the improper tools, damage can occur(e.g., especially damage to the heart and/or lungs). In someembodiments, implantation of the substernal lead of the extravascularICD system involves using a tunneling tool to create a route through thesubsternal space for the lead 706. For example, a tunneling tool can beused to form the route from the xiphoid process 710 underneath thesternum 702 in the substernal space. A tunneling tool can also be usedto create a route from the sternum 702 to the left midaxillary side ofthe patient above the ribcage when the ICD 704 is implanted (e.g., belowthe left armpit). After the substernal route is established, thetunneling tool is removed and the lead is fed through the substernalroute.

With reference to FIGS. 1, 5, 6 and 7, in accordance with variousembodiments described herein, instrument 106 is an example tunnelingtool that can be used to create the route for the extravascular lead706. The tunneling tool can include a stainless steel cylindrical tubefeaturing an atraumatic distal tip. To create the route, the tunnelingtool is inserted (e.g., manually or using a robotic arm) into the body102 of the patient via an incision (e.g., incision 124 and/or incision122). The tunneling tool is then advanced through the body 102 of thepatient to a target area or end point beneath the sternum 702 and withinthe anterior mediastinum.

When creating the route through the substernal space, the tunneling tool(e.g., instrument 106) can be carefully maneuvered within the body 102along a defined route so as to reduce the likelihood of damaging thenerves in the auxiliary region, piercing the lungs, piercing the heart,perforating muscles, routing the tunneling tool between muscle layers,etc. When properly inserted, the tunneling tool can pass under the skinand over the muscles in the torso of the body 102 of the patient to atarget area and/or along a target path within the body 102 of thepatient.

In accordance with various embodiments, system 100 and the like (e.g.,system 500 and system 600) can be configured to facilitate the abovedescribed tunneling procedure using the disclosed augmented realitytechniques to visualize the tunneling tool relative to internalanatomical structures during insertion and/or advancement of thetunneling tool through the body. For example, a model of the internalanatomical structures within the chest and torso can initially begenerated based on previously captured imaging data (e.g., MRI data,X-ray data, fluoroscopy data, etc.) and known structure and componentsof the patient anatomy (e.g., the human anatomy). The model can includethe specific internal anatomical structures that the tunneling toolshould avoid as well as the internal anatomical structures that thetunneling tool should be advanced near. In some embodiments, theinternal anatomical structures that the tunneling tool should avoid canbe visually distinguished (e.g., by a different color or pattern) in thevisualization presented via display component 118 from the internalanatomical structures that the tunneling tool should be advanced near.

As the tunneling tool is inserted into the body 102 of the patient viaincision 122 and/or 124, the precise position and orientation of thetunneling tool relative to the modeled anatomical structures can bepresented to the user via the visualization generated by imaging device116 and presented via display component 118, in accordance with theaspects and embodiments discussed herein. In some instances, when thetunneling tool is inserted via incision 122 to form a subcutaneous routefrom the left midaxillary region toward the sternum 702, the tunnelingtool can be felt under the skin of the patient. Accordingly, in additionto an augmented reality visualization of the tool provided by system 100and the like (e.g., system 500 and system 600), a user can manually feelthe location of the tool underneath the skin of the patient as the toolis advanced toward the sternum to facilitate guiding the tool to theproper position. However, when forming a route from the xiphoid process710 (e.g., via incision 124) and beneath the sternum 702 into thesubsternal space, the sternum 702 and ribcage block the ability of theuser to externally feel and guide the tool. Accordingly, by generatingand presenting the user with an augmented reality visualization of thetool as it is advanced underneath the sternum 702, the user is able tomore accurately form the route for the substernal lead 706 and reducethe likelihood of complications.

In an exemplary embodiment, a sternal fiducial marker 104 can be placedon the body 102 of the patient directly above the sternum 702 and/or atthe location where the distal end of the lead 706 should be placed. Thissternal fiducial marker 104 can serve as a reference point/object suchthat the position and orientation of the in-vivo portion of thetunneling tool can be directly correlated to a known reference point(e.g., the position of the sternal fiducial marker 104 relative to a 2-Dor 3-D coordinate space can be predetermined) directly above the sternumand/or target location for the distal end of the lead 706. In otherembodiments, another incision fiducial marker 120 can be placed on thebody 102 of the patient at a known position near the incision 122(and/or incision 124) where the tunneling tool is inserted. Accordingly,the incision fiducial marker 120 can serve as another referencepoint/object such that the position and orientation of the in-vivoportion of the tunneling tool can be directly correlated to a knownreference point at the incision 122.

In another embodiment, the tunneling tool (e.g., instrument 106) caninclude a set of electrode marks/fiducial markers on a portion of theshaft 112 that corresponds to the locations within the body 102 wherethe respective electrodes 712, 714, and 716 of the lead 706 should beplaced. For example, FIG. 8 illustrates a schematic diagram of anotherexample, non-limiting instrument 106 that facilitates surgery usingaugmented reality in accordance with one or more embodiments describedherein. Repetitive description of like elements described withrespective embodiments disclosed herein is omitted for sake of brevity.

With reference to FIGS. 1, 5, 6, 7 and 8, instrument 106 can include aset of electrode markers 812, 814, and 816 located on distal portion ofthe shaft 112. These electrode markers 812, 814 and 816 can becorrelated to locations and/or dimensions of the respective electrodes712, 714, and 716 of the lead 706. Prior to insertion of the instrument106 into the patient, the geometry of the instrument can be determined,including the spatial relationships between the respective electrodemarkers 812, 814, and 816 to one another and the geometrical/spatialrelationships between the respective electrode markers 812, 814 and 816and other points on the instrument that are configured to remain outsidethe body. For example, the imaging device 116 can capture image data ofthe instrument 106 and determine the spatial/geometrical relationshipsbetween the respective markers to one another and fiducial marker 200.

In accordance with this embodiment, when the distal end of the shaft 112is inserted into the body 102 of the patient, the imaging device 116 candetermine the location of the respective electrode markers 812, 814 and816 relative to the internal anatomical features of the body 102 basedon the predefined geometrical/spatial relationships determined betweenthe electrode markers 812, 814 and 816, and fiducial marker 200 (and/orother points on the instrument 106 configured to remain outside of thebody), and a predetermined geometrical/spatial relationship betweenfiducial marker 200 relative to one or more reference points/markers notlocated on the instrument (e.g., a body part, an external fiducialmarker such as fiducial marker 104, etc.). As a result, imaging device116 can generate a visualization depicting the locations of therespective electrode markers 812, 814 and 816 beneath the sternum 702 tofacilitate aligning the electrode markers 812, 814 and 816 with thecorrect anatomical features within the body 102 where the electrodes712, 714 and 716 of the lead 706 should be placed.

Although various embodiments of augmented reality assisted surgery aredescribed herein in connection with performing extravascular ICDimplantation, it should be appreciated that the disclosed embodimentsare not limited to surgery involving extravascular ICD implantation. Forexample, the disclosed embodiments can be applied to various types ofsurgical procedures involving an instrument 106 that is inserted intothe body, especially procedures in which cutting or opening of the body102 of the patient is employed to visualize the surgical area beingtreated or addressed. In addition, it should be noted that the disclosedsystems, methods, and computer readable media may not be limited totreatment of a human patient. In alternative examples, the disclosedembodiments can be implemented in non-human patients, e.g., primates,canines, equines, pigs, ovines, bovines, and felines. For example, theseanimals can undergo clinical or research therapies that may benefit fromthe subject matter of this disclosure. Finally, the disclosedembodiments can be used to assist actual surgery, exploratory surgery,or simulated surgery performed on non-live or model cadavers fortraining and procedural development purposes.

FIGS. 9-11 illustrate schematic diagrams of example, non-limitingsystems 900, 1000, and 1100 that facilitate surgery using augmentedreality in accordance with one or more embodiments described herein.Systems 900, 1000, and 1100 demonstrate the features and functionalitiesof systems 100, 500, 600. Systems 900, 1000, and 1100 include examplerespective visualizations 902, 1002 and 1102 generated by imaging device116 and/or rendered via display component 118 based on image datacaptured via image capture device 114 and/or image capture device 1004during insertion of instrument 106 into the patient via incision 124and/or 122, in accordance with the various embodiments disclosed herein.The example visualizations respectively include internal structures ofthe patient 102 relative to a current position and orientation of aportion of the instrument 106. Repetitive description of like elementsemployed in respective embodiments is omitted for sake of brevity.

With reference to FIG. 9, illustrated is an example system 900 showing avisualization 902 of a body 102 of a patient, as depicted to the left ofthe visualization 902, with an instrument 106 inserted into incision 124and incision 122. Visualization 902 provides a top down perspective ofinternal structures of the chest/torso with the instrument 106 inserted.In an exemplary embodiment, visualization 902 is a 3-D model. In otherembodiments, visualization 902 can be depicted as a 2-D model. As shownin visualization 902, the in-vivo portions of the instrument 106relative to internal structures of the body and incisions 124 and 122are presented.

FIG. 10 illustrates another example visualization 1002 of an actual body102 of a patient, as depicted to the left of the visualization 1002,with an instrument 106 inserted into incision 124. Visualization 1002provides a side view perspective of internal structures of thechest/torso with instrument 106 inserted. To facilitate generating thisside view perspective, system 1000 can include image capture device 114positioned above the patient and pointed substantially perpendicularrelative to the operating area, and another image capture device 1004positioned on a side of the patient and substantially coplanar with theoperating area. Image capture device 1004 can include the features andfunctionality of image capture device 114 as described with respect toFIG. 1. In an exemplary embodiment, visualization 1002 is a 3-D model.In other embodiments, visualization 1002 can be depicted as a 2-D model.As shown in visualization 1002, the in-vivo portion of the instrument106 relative to internal structures of the body and incision 124 ispresented.

FIG. 11 illustrates another system 1100 showing a visualization 1102 ofa body 102 of a patient, as depicted to the left of the visualization1102, with an instrument 106 inserted into incision 122. Visualization1102 provides a cross-sectional or transverse view of internalstructures of the chest/torso with instrument 106 inserted. Tofacilitate generating this transverse view perspective, system 1100 caninclude image capture device 114 positioned above the patient andpointed substantially perpendicular relative to the operating area, andanother image capture device 804 positioned on a side of the patient andsubstantially coplanar with the operating area. In an exemplaryembodiment, visualization 1102 is a 2-D model. For example, in anexemplary embodiment, visualization 1102 is an orthographic projectionof a transverse cross-section of the chest at a position above thesternal space. In other embodiments, visualization 1102 can be depictedas a 3-D model. As shown in visualization 1102, the in-vivo portion ofthe instrument 106 relative to internal structures of the body ispresented (e.g., as the black dot).

FIG. 12 illustrates a block diagram of an example, non-limiting imagingdevice facilitating surgery using augmented reality in accordance withone or more embodiments described herein. Repetitive description of likeelements employed in other embodiments described herein is omitted forsake of brevity.

In addition to display component 118, imaging device 116 can includereception component 1202, positioning component 1204, guidance component1206, rendering component 1210, memory 1212, speaker 1214, and processor1216. Memory 1212 can store computer executable components, andprocessor 1216 that executes the computer executable components storedin the memory (e.g., reception component 1202, positioning component1204, guidance component 1206, rendering component 1210, and softwarecomponents of display component 1218). Imaging device 116 can furtherinclude a bus 1208 that couples the various components including, butnot limited to, reception component 1202, positioning component 1204,guidance component 1206, rendering component 1210, display component1218, memory 1212, speaker 1214, and processor 1216.

Reception component 1202 can be configured to receive image datacaptured by one or more image capture devices (e.g., image capturedevice 114 and the like) and orientation data captured via anorientation unit 126 attached to instrument 106. Reception component1202 can also receive predetermined, known and/or definedinformation/data that facilitates determining the position andorientation of an in-vivo portion of an instrument and generatingvisualizations of the instruments relative to anatomical structures ofthe patient. For example, reception component 1202 can receivepredetermined, defined, known or preconfigured virtual models of apatient, defined coordinate positions of one or more referenceobjects/points relative to a 2-D or 3-D coordinate space correspondingto the operating environment, information identifying fiducial markerpatterns, information identifying the geometry, size and/or shape of ainstrument 106 to be tracked during a surgical procedure, etc.Information received by reception component 1202 can be processed by theimaging device 116 and/or stored in memory 1212.

Positioning component 1204 can be configured to determine positions andorientations of an in-vivo portion of an instrument 106 relative to oneor more anatomical features of the patient based on the informationreceived by reception component 1202, as described in detail withrespect to FIG. 1. For example, positioning component 1204 can determinerelative positions of an internal portion of the instrument insertedinto a body 102 of the patient relative to one or more anatomicalstructures of the body 102 based on a known or determined size and shapeof the instrument and image data including, but not limited to, afiducial marker or portion of the fiducial marker on the instrument, anexternal part of the body, another fiducial marker located on anexternal part of the body or the like.

Rendering component 1210 can be configured to generate 2-D and/or 3-Dvisualizations including a representation of the instrument 106 at adetermined position and orientation of the instrument 106 relative toone or more modeled anatomical structures of the body corresponding toan actual position and orientation of the instrument 106 relative to theactual anatomical structures of the body 102. For example, renderingcomponent 1210 can generate and/or configure visualizations 902, 1002and 1102. The visualizations generated or configured by renderingcomponent 1210 can further be displayed or presented via a graphicaluser interface, or as a hologram by, display component 118.

Guidance component 1206 can be configured to provide guidance to a userduring insertion and/or maneuvering of an instrument 106 within a body102 of a patient based on the determined positions and orientations ofthe instrument 106 relative to internal anatomical structures of thebody, and/or based on previously determined parameters or metricsregarding how and where the instrument 106 should be inserted,maneuvered, oriented and positioned during the particular procedurebeing performed. For example, in an embodiment directed to employinginstrument 106 to create a route for insertion of the extravascular lead706, guidance component 1206 can be provided access to information(e.g., in memory 1212 or at another device) identifying an optimal routevia which the instrument 106 should advance within the body 102 of thepatient. For example, the information can identify respective pointsalong a 2-D or 3-D coordinate space corresponding to the internalstructure of the body 102 of the patient. The information can alsoidentify precise distances (e.g., in centimeters or millimeters)relative to certain internal anatomical structures from which theinstrument 106 should be located at respective points along the route.The information can also identify internal structures that theinstrument 106 should avoid contacting and/or distances that theinstrument 106 should be located from relative to these internalstructures during insertion into the body 102.

In an embodiment, guidance component 1206 can track the positions andorientations of the instrument 106 as it is inserted to determine atrajectory path of the instrument 106. According to this embodiment,guidance component 1206 can compare a current trajectory path for theinstrument 106 with an optimal route for the instrument 106 to determinewhether the instrument 106 is on the proper track. In response to adetermination that the instrument 106 is off course, the guidancecomponent 1206 can provide the user with instruction that facilitatesaligning the instrument 106 back on the proper track. For example,guidance component 1206 can direct rendering component 1210 to draw avirtual line on a visualization corresponding to the correct track andanother virtual line corresponding to the current off target trajectoryof the instrument 106.

In another example, guidance component 1206 can determine how to adaptthe current position and/or orientation of the instrument 106 to bringthe instrument 106 onto the proper track. For instance, guidancecomponent 1206 can determine that the instrument 106 should move adefined number of centimeters to the right or left, and/or anotherdefined number of centimeters up or down, etc.

The guidance component 1206 can further provide these instructions tothe user in a visual format (e.g., as text, arrows, or other visual cuespresented with the visualization) and/or an audible format (e.g., asspoken instructions output via the speaker 1214). Guidance component1206 can also determine when the instrument 106 is near or headingtoward an anatomical feature that should be avoided by the instrument106 (e.g., the heart, the ribs, etc.). The guidance component 1206 canfurther provide instruction notifying the user of the incorrect anddangerous trajectory to prevent injury to the patient.

FIG. 13 illustrates a schematic diagram of an example, non-limitingvisualization 1300 that facilitates surgery using augmented reality inaccordance with one or more embodiments described herein. In someembodiments, visualization 1300 can include the same or similar featuresas visualization 1002. Repetitive description of like elements employedin respective embodiments disclosed herein is omitted for sake ofbrevity.

Visualization 1300 can illustrate various visual guidance featuresafforded by guidance component 1206. In particular, visualization 1300can include a first trajectory line 1302 extending from the distal endof the instrument 106 indicating the current trajectory path of theinstrument 106. Visualization 1300 can also include a second trajectoryline 1304 corresponding to the correct or optimal trajectory path foradvancement of the instrument 106. By looking at visualization 1300, theuser operating the instrument 106 can clearly see that the instrument106 is off course and maneuver the instrument 106 to the correct course(e.g., second trajectory line 1304).

Visualization 1300 can also include a target mark 1310 that indicates atarget end position for the instrument 106. For example, with respect toformation of a substernal route for implantation of an extravascularlead, the target mark 1310 is located below the sternum 1308 within anarea of the anterior mediastinum 1312 between the sternum 1308 and theheart 1306. In some embodiments, guidance component 1206 can furtherprovide visual and/or audible instruction informing the user ofpositions (e.g., in actual centimeters, millimeters or any other unitsthat are appropriate) the instrument 106 is located relative to targetmark 1310 and the identified anatomical structures.

FIGS. 14A and 14B illustrate schematic diagrams of example, non-limitinginstruments that facilitate surgery using augmented reality inaccordance with additional embodiments described herein. Repetitivedescription of like elements employed in respective embodimentsdisclosed herein is omitted for sake of brevity.

Similar to previously described embodiments, the instrument 106 depictedin FIGS. 14A and 14B includes an elongated shaft 112 configured toinsert inside the body of a patient. The elongated shaft 112 can have acurved geometry to facilitate forming a substernal route in someimplementations. In accordance with the embodiments depicted in FIGS.14A and 14B, the handle 110 previously described (e.g., in FIG. 1) hasbeen replaced with a handgrip 1402 that facilitates mechanicallycontrolling the position, orientation and/or configuration of theelongated shaft 112. For example, as depicted in FIG. 14A, in oneembodiment, the elongated shaft 112 can have one or more articulationjoints (e.g., articulation joints 1410 and 1412) that allow portions orsections of the elongated shaft separated by the articulation joints tomechanically pivot or rotate relative to one another. The manner ofmovement enabled by the articulation joints can vary. For example, thearticulation joints 1410 and 1412 can include, but are not limited to, aball and socket joint, a hinge joint, a condyloid joint (a joint thatpermits all forms of angular movement except axial rotation), a pivotjoint, gliding joint, and/or a saddle joint.

In another example, as depicted in FIG. 14B, the length of the elongatedshaft 112 can be dynamically adapted via mechanical movement ofdifferent segments (e.g., segments 1414 and 1418) of the elongated shaft112 relative to one another. For instance, the elongated shaft 112 caninclude two or more attached segments 1414 and 1418, wherein at leastone of the segments is configured to insert into the other and extendaway from and retract back into the other segment in a telescope manner.For example, segment 1418 is depicted with a smaller radius than segment1414 and extending outward from segment 1414. According to this example,segment 1418 is configured to be able to further extend a defined lengthoutward from segment 1414 and/or retract back into segment 1414.

In one or more implementations, the handgrip 1402 can include one ormore manual control input components (e.g., control button 1404, triggerbutton 1406, joystick control 1408) that can facilitate manuallycontrolling movement and/or configuration of the elongated shaft 112based on the mechanical configuration of the elongated shaft enabled bythe articulation joints (e.g., articulation joints 1410 and 1412) and/orthe retractable segments (e.g., segment 1418). For example, theinstrument 106 can include a motor (not shown) and electrical circuitry(not shown) coupling the one or more input components, the motor, and/orthe different segments and/or joints of the elongated shaft 112. Themotor can be configured to respond to control instruction input receivedfrom the one or more manual input control components to cause theelongated shaft 112 to move in a manner defined by the controlinstruction input. For example, with reference to FIG. 14A, the triggerbutton 1406 can be configured to direct a segment of the elongated shaft112 to move up relative to an articulation joint (e.g., articulationjoint 1410 or 1412), the control button 1404 can be configured to directthe segment to move down relative to the articulation joint, and/or thejoystick control 1408 can be configured to direct the segment to moveleft and right relative to the articulation joint. In another example,with reference to FIG. 14B, the trigger button 1406 can be configured todirect segment 1418 to extend outward from segment 1414 and/or thecontrol button 1404 can be configured to direct segment 1418 to retractinto segment 1414.

With reference to FIGS. 14A, 14B, 1 and 10, in various embodiments, theinstrument 106 can be communicatively coupled to imaging device 116,either via one or more wires or wirelessly. The instrument 106 can alsoinclude a power source to facilitate mechanical and electrical operationof the instrument 106, and/or be coupled to an external power source viaone or more wires (e.g., a power source provided by imaging device 116and/or another device). The instrument 106 can be further configured toprovide the imaging device 116 with the control input information thatdefines the input control instructions applied to the instrument via theone or more input control components (e.g., control button 1404, triggerbutton 1406 and joystick control 1408).

According to these embodiments, the imaging device 116 can be configuredto receive the control input information (e.g., via reception component1202) and map the control input information to a configuration of theelongated shaft 112. For example, the positioning component 1204, candetermine a current configuration of the elongated shaft 112 relative toa default configuration of the elongated shaft 112 based on the changesto the configuration of the elongated shaft 112 caused by the controlinput information. The positioning component 1204 can further determinea position and/or configuration of the instrument 106 and the elongatedshaft 112 relative to the body 102 of the patient and one or moreinternal anatomical features of the patient based on the determinedconfiguration of the elongated shaft 112, a position and/or orientationof the handgrip 1402 portion of the instrument 106 that is outside ofthe body 102 (e.g., using the various imaging positioning techniquesdescribed herein) and/or a known configuration of the entire instrument106 (e.g., a known configuration of the handgrip 1402 portion relativeto the elongated shaft 112 portion). The rendering component 1210 canthen generate 2-D and/or 3-D visualizations of the elongated shaft 112at the determined position, orientation and/or configuration inside thebody 102 relative to the one or more anatomical features as, previouslydescribed.

In view of the example systems and/or devices described herein, examplemethods that can be implemented in accordance with the disclosed subjectmatter can be further appreciated with reference to flowcharts in FIGS.15-17. Repetitive description of like elements employed in respectiveembodiments is omitted for sake of brevity.

For purposes of simplicity of explanation, example methods disclosedherein are presented and described as a series of acts; however, it isto be understood and appreciated that the disclosed subject matter isnot limited by the order of acts, as some acts may occur in differentorders and/or concurrently with other acts from that shown and describedherein. For example, a method disclosed herein could alternatively berepresented as a series of interrelated states or events, such as in astate diagram. Moreover, interaction diagram(s) may represent methods inaccordance with the disclosed subject matter when disparate entitiesenact disparate portions of the methods. Furthermore, not allillustrated acts may be required to implement a method in accordancewith the subject specification. It should be further appreciated thatthe methods disclosed throughout the subject specification are capableof being stored on an article of manufacture to facilitate transportingand transferring such methods to computers for execution by a processoror for storage in a memory.

FIG. 15 illustrates a flow diagram of an example, non-limiting method1500 facilitating surgery using augmented reality in accordance with oneor more embodiments described herein. At 1502, a device (e.g., imagingdevice 116) including a processor can receive image data associated withan external portion of a tool located within a body of a patient. Theimage data can include first information indicative of a first fiducialmarker on the external portion of the tool. At 1504, the device candetermine one or more relative positions of an internal portion of thetool within the body relative to one or more anatomical structures ofthe body based on the image data and a defined configuration (e.g.,known size and shape) of the tool. At 1506, the device can generate oneor more representations of the tool within the body relative to the oneor more anatomical structures based on the one or more relativepositions and the defined configuration of the tool. For example, therepresentations can respectively correspond to different positions,orientation, and perspectives of the tool relative to the one or moreanatomical structures over the course of surgery.

FIG. 16 illustrates a flow diagram of another example, non-limitingmethod 1600 facilitating surgery using augmented reality in accordancewith one or more embodiments described herein. At 1602, a device (e.g.,imaging device 116) including a processor can receive image dataassociated with an external portion of an instrument (e.g., instrument106) located within a body of a patient. The image data can includefirst information indicative of a first fiducial marker on the externalportion of the instrument. At 1604, the device can determine one or morerelative positions of an internal portion of the instrument within thebody relative to one or more anatomical structures of the body based onthe image data and a defined configuration of the instrument. At 1606,the device can generate one or more representations of the instrumentwithin the body relative to the one or more anatomical structures basedon the relative positions and the defined configuration of theinstrument. For example, the representations can respectively correspondto different positions, orientation, and perspectives of the instrumentrelative to the one or more anatomical structures over the course ofsurgery. At 1608, the device can display (e.g., via display component118) the one or more representations on a first display of the device.The one or more representations can facilitate guidance of theinstrument to a target location within the body.

FIG. 17 illustrates a flow diagram of another example, non-limitingmethod 1700 facilitating surgery using augmented reality in accordancewith one or more embodiments described herein. At 1702, a device (e.g.,imaging device 116) including a processor can receive image dataassociated with an external portion of an instrument (e.g., instrument106) as it is inserted into a body of a patient. The image data caninclude at least a fiducial marker on the external portion of theinstrument. At 1704, the device can determine relative positions of aninternal portion of the instrument within the body relative to one ormore anatomical structures of the body based on the image data and aknown size and shape of the instrument.

At 1706, the device can generate representations of the instrumentwithin the body relative to the one or more anatomical structures basedon the relative positions and the known size and shape of theinstrument. For example, the representations can respectively correspondto different positions, orientation, and perspectives of the instrumentrelative to the one or more anatomical structures over the course ofsurgery.

At 1708, the device can determine a trajectory of path of the instrumentwithin the body based on the relative positions of the instrument withinthe body relative to the one or more anatomical structures. At 1710, thedevice can determine an error in a current position of the instrumentwithin the body based on the trajectory path and a target location forthe instrument within the body. At 1712, the device can determine achange to the current position of the instrument based on the error, andat 1714, the device can output information associated with aninstruction regarding the change (e.g., visual and/or audibleinstruction) to the current position of the instrument.

FIG. 18 illustrates a block diagram of an example, non-limitingenvironment including a computer operable to facilitate surgery usingaugmented reality in accordance with one or more embodiments describedherein. FIG. 18 and the following discussion are intended to provide abrief, general description of a suitable computing environment 1800 inwhich the one or more embodiments described herein can be implemented.The computer 1802 can be or include the image capture device 114,imaging device 116, display component 118, orientation unit 126 and/ordevice 502 in accordance with one or more embodiments described herein.Repetitive description of like elements employed in other embodimentsdescribed herein is omitted for sake of brevity.

Generally, program modules include routines, programs, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the inventive methods can be practiced with other computer systemconfigurations, including single-processor or multiprocessor computersystems, minicomputers, mainframe computers, as well as personalcomputers, hand-held computing devices, microprocessor-based orprogrammable consumer electronics, and the like, each of which can beoperatively coupled to one or more associated devices.

Computing devices typically include a variety of media, which caninclude computer-readable storage media and/or communications media,which two terms are used herein differently from one another as follows.Computer-readable storage media can be any available storage media thatcan be accessed by the computer and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable storage media can be implementedin connection with any method or technology for storage of informationsuch as computer-readable instructions, program modules, structured dataor unstructured data. Tangible and/or non-transitory computer-readablestorage media can include, but are not limited to, random access memory(RAM), read only memory (ROM), electrically erasable programmable readonly memory (EEPROM), flash memory or other memory technology, compactdisk read only memory (CD-ROM), digital versatile disk (DVD) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage, other magnetic storage devices and/or other media that can beused to store desired information. Computer-readable storage media canbe accessed by one or more local or remote computing devices, e.g., viaaccess requests, queries or other data retrieval protocols, for avariety of operations with respect to the information stored by themedium.

In this regard, the term “tangible” herein as applied to storage,memory, computer-readable media or computer-readable storage media, isto be understood to exclude only propagating intangible signals per seas a modifier and does not relinquish coverage of all standard storage,memory, computer-readable media or computer-readable storage media thatare not only propagating intangible signals per se.

In this regard, the term “non-transitory” herein as applied to storage,memory, computer-readable media or computer-readable storage media, isto be understood to exclude only propagating transitory signals per seas a modifier and does not relinquish coverage of all standard storage,memory, computer-readable media or computer-readable storage media thatare not only propagating transitory signals per se.

Communications media typically embody computer-readable instructions,data structures, program modules or other structured or unstructureddata in a data signal such as a modulated data signal, e.g., a channelwave or other transport mechanism, and includes any information deliveryor transport media. The term “modulated data signal” or signals refersto a signal that has one or more of the data signal's characteristicsset or changed in such a manner as to encode information in one or moresignals. By way of example, and not limitation, communication mediainclude wired media, such as a wired network or direct-wired connection,and wireless media such as acoustic, RF, infrared and other wirelessmedia.

With reference again to FIG. 18, example environment 1800 that can beemployed to implement one or more embodiments of the embodimentsdescribed herein includes computer 1802. Computer 1802 includesprocessing unit 1804, system memory 1806 and system bus 1808. System bus1808 couples system components including, but not limited to, systemmemory 1806 to processing unit 1804. Processing unit 1804 can be any ofvarious commercially available processors. Dual microprocessors andother multi-processor architectures can also be employed as processingunit 1804.

System bus 1808 can be any of several types of bus structure that canfurther interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. System memory 1806 includesRAM 1810 and ROM 1812. A basic input/output system (BIOS) can be storedin a non-volatile memory such as ROM, erasable programmable read onlymemory (EPROM), EEPROM, which BIOS contains the basic routines that helpto transfer information between elements within computer 1802, such asduring startup. RAM 1810 can also include a high-speed RAM such asstatic RAM for caching data.

Computer 1802 further includes internal hard disk drive (HDD) 1814(e.g., Enhanced Integrated Drive Electronics (EIDE), Serial AdvancedTechnology Attachment (SATA)). HDD 1814 can be connected to system bus1808 by hard disk drive interface 1816. The drives and their associatedcomputer-readable storage media provide nonvolatile storage of data,data structures, computer-executable instructions, and so forth. Forcomputer 1802, the drives and storage media accommodate the storage ofany data in a suitable digital format.

A number of program modules can be stored in the drives and RAM 1810,including operating system 1836, one or more application programs 1838,other program modules 1840 and program data 1842. All or portions of theoperating system, applications, modules, and/or data can also be cachedin RAM 1810. The systems and methods described herein can be implementedutilizing various commercially available operating systems orcombinations of operating systems.

A mobile device can enter commands and information into computer 1802through one or more wireless input devices, e.g., wireless keyboard 1828and a pointing device, such as wireless mouse 1830. Other input devices(not shown) can include a smart phone, tablet, laptop, wand, wearabledevice or the like. These and other input devices are often connected tothe processing unit 1804 through input device interface 1818 that can becoupled to system bus 1808, but can be connected by other interfaces,such as a parallel port, an IEEE serial port, a game port and/or auniversal serial bus (USB) port.

Computer 1802 can operate in a networked environment using logicalconnections via wired and/or wireless communications to one or moreremote computers, such as remote computer(s) 1832. Remote computer(s)1832 can be a workstation, a server computer, a router, a personalcomputer, portable computer, microprocessor-based entertainmentappliance, a peer device or other common network node, and typicallyincludes many or all of the elements described relative to computer1802, although, for purposes of brevity, only memory/storage device 1834is illustrated. The logical connections depicted include wired/wirelessconnectivity to a local area network (LAN) 1826 and/or larger networks,e.g., WAN 1824, as well as smaller PANs involving a few devices (e.g.,at least two). LAN and WAN networking environments are commonplace inthe home, offices (e.g., medical facility offices, hospital offices) andcompanies, and facilitate enterprise-wide computer networks, such asintranets, all of which can connect to a global communications network(e.g., the Internet).

When used in a LAN networking environment, computer 1802 can beconnected to local network through a wired and/or wireless communicationnetwork interface or adapter 1820. Adapter 1820 can facilitate wired orwireless communication to LAN 1826, which can also include a wirelessaccess point (AP) connected to the LAN 1826 for communicating withadapter 1820.

When used in a WAN networking environment, computer 1802 can includemodem 1822 or can be connected to a communications server on WAN 1824 orhas other means for establishing communications over WAN 1824, such asby way of the Internet. Modem 1822, which can be internal or externaland a wired or wireless device, can be connected to system bus 1808 viainput device interface 1816. In a networked environment, program modulesdepicted relative to computer 1802 or portions thereof, can be stored ina remote memory/storage device. It will be appreciated that the networkconnections shown are example and other means of establishing acommunications link between the computers can be used.

Computer 1802 can be operable to communicate with any wireless devicesor entities operatively disposed in wireless communication via anynumber of protocols, including, but not limited to, NFC, Wi-Fi and/orBLUETOOTH® wireless protocols. Thus, the communication can be a definedstructure as with a conventional network or simply an ad hoccommunication between at least two devices.

NFC can allow point-to-point connection to an NFC-enabled device in theNFC field of an IMD within the home or at any location. NFC technologycan be facilitated using an NFC-enabled smart phone, tablet or otherdevice that can be brought within 3-4 centimeters of an implanted NFCcomponent. NFC typically provides a maximum data rate of 424 kilobitsper second (Kbps), although data rates can range from 6.67 Kbps to 828Kbps. NFC typically operates at the frequency of 13.56 megahertz (MHz).NFC technology communication is typically over a range not exceeding 0.2meters (m) and setup time can be less than 0.1 seconds. Low power (e.g.,18 milliamperes (mAs)) reading of data can be performed by an NFCdevice.

Wi-Fi can allow connection to the Internet from a couch at home, a bedin a hotel room or a conference room at work, without wires. Wi-Fi is awireless technology similar to that used in a cell phone that enablessuch devices, e.g., computers, to send and receive data indoors and out.Wi-Fi networks use radio technologies called IEEE 802.11 (a, b, g, n,etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Finetwork can be used to connect computers to each other, to the Internet,and to wired networks (which can use IEEE 802.3 or Ethernet). Wi-Finetworks operate in the unlicensed 2.4 and 5 GHz radio bands, at an 18Mbps (802.11a) or 54 Mbps (802.11b) data rate, for example or withproducts that contain both bands (dual band), so the networks canprovide real-world performance similar to the basic 10BaseT wiredEthernet networks used in many offices.

The embodiments of devices described herein can employ artificialintelligence (AI) to facilitate automating one or more featuresdescribed herein. The embodiments (e.g., in connection withautomatically identifying acquired cell sites that provide a maximumvalue/benefit after addition to an existing communication network) canemploy various AI-based schemes for carrying out one or more embodimentsthereof. Moreover, the classifier can be employed to determine a rankingor priority of each cell site of an acquired network. A classifier is afunction that maps an input attribute vector, x=(x1, x2, x3, x4, . . . ,xn), to a confidence that the input belongs to a class, that is,f(x)=confidence(class). Such classification can employ a probabilisticand/or statistical-based analysis (e.g., factoring into the analysisutilities and costs) to prognose or infer an action that a mobile devicedesires to be automatically performed. A support vector machine (SVM) isan example of a classifier that can be employed. The SVM operates byfinding a hypersurface in the space of possible inputs, which thehypersurface attempts to split the triggering criteria from thenon-triggering events. Intuitively, this makes the classificationcorrect for testing data that is near, but not identical to trainingdata. Other directed and undirected model classification approachesinclude, e.g., naïve Bayes, Bayesian networks, decision trees, neuralnetworks, fuzzy logic models, and probabilistic classification modelsproviding different patterns of independence can be employed.Classification as used herein also is inclusive of statisticalregression that is utilized to develop models of priority.

As will be readily appreciated, one or more of the embodiments canemploy classifiers that are explicitly trained (e.g., via a generictraining data) as well as implicitly trained (e.g., via observing mobiledevice behavior, operator preferences, historical information, receivingextrinsic information). For example, SVMs can be configured via alearning or training phase within a classifier constructor and featureselection module. Thus, the classifier(s) can be used to automaticallylearn and perform a number of functions, including but not limited todetermining according to a predetermined criteria which of the acquiredcell sites will benefit a maximum number of subscribers and/or which ofthe acquired cell sites will add minimum value to the existingcommunication network coverage, etc.

As employed herein, the term “processor” can refer to substantially anycomputing processing unit or device including, but not limited to,single-core processors; single-processors with software multithreadexecution capability; multi-core processors; multi-core processors withsoftware multithread execution capability; multi-core processors withhardware multithread technology; parallel platforms; and parallelplatforms with distributed shared memory. Additionally, a processor canrefer to an integrated circuit, an application specific integratedcircuit (ASIC), a digital signal processor (DSP), a field programmablegate array (FPGA), a programmable logic controller (PLC), a complexprogrammable logic device (CPLD), a discrete gate or transistor logic,discrete hardware components or any combination thereof designed toperform the functions described herein. Processors can exploitnano-scale architectures such as, but not limited to, molecular andquantum-dot based transistors, switches and gates, in order to optimizespace usage or enhance performance of mobile device equipment. Aprocessor can also be implemented as a combination of computingprocessing units.

Memory disclosed herein can include volatile memory or nonvolatilememory or can include both volatile and nonvolatile memory. By way ofillustration, and not limitation, nonvolatile memory can include ROM,programmable ROM (PROM), electrically programmable ROM (EPROM),electrically erasable PROM (EEPROM) or flash memory. Volatile memory caninclude RAM, which acts as external cache memory. By way of illustrationand not limitation, RAM is available in many forms such as static RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), anddirect Rambus RAM (DRRAM). The memory (e.g., data storages, databases)of the embodiments is intended to include, without being limited to,these and any other suitable types of memory.

As used herein, terms such as “data storage,” “database,” andsubstantially any other information storage component relevant tooperation and functionality of a component, refer to “memorycomponents,” or entities embodied in a “memory” or components includingthe memory. It will be appreciated that the memory components orcomputer-readable storage media, described herein can be either volatilememory or nonvolatile memory or can include both volatile andnonvolatile memory.

In addition, the words “example” and “exemplary” are used herein to meanserving as an instance or illustration. Any embodiment or designdescribed herein as “example” or “exemplary” is not necessarily to beconstrued as preferred or advantageous over other embodiments ordesigns. Rather, use of the word “example” or “exemplary” is intended topresent concepts in a concrete fashion. As used in this application, theterm “or” is intended to mean an inclusive “or” rather than an exclusive“or”. That is, unless specified otherwise or clear from context, “Xemploys A or B” is intended to mean any of the natural inclusivepermutations. That is, if X employs A; X employs B; or X employs both Aand B, then “X employs A or B” is satisfied under any of the foregoinginstances. In addition, the articles “a” and “an” as used in thisapplication should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. The terms “first,” “second,” “third,” and so forth, as used in theclaims and description, unless otherwise clear by context, is forclarity only and doesn't necessarily indicate or imply any order intime.

What has been described above includes mere examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing these examples, but one of ordinary skill in the art canrecognize that many further combinations and permutations of the presentembodiments are possible. Accordingly, the embodiments disclosed and/orclaimed herein are intended to embrace all such alterations,modifications and variations that fall within the spirit and scope ofthe detailed description and the appended claims. Furthermore, to theextent that the term “includes” is used in either the detaileddescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprising” as “comprising” is interpretedwhen employed as a transitional word in a claim.

1. A medical device comprising: an elongated shaft comprising a distalend and a proximal end, wherein the distal end is configured forinsertion into a body of a patient; a handle coupled to the proximal endof the elongated shaft; one or more fiducial markers located on theelongated shaft; and an orientation unit comprising an accelerometerconfigured to collect orientation information regarding an orientationof the medical device and provide the orientation information to anotherdevice, wherein an orientation and position of the medical devicerelative to the body of the patient are determined based on theorientation information and image data captured by one of the one ormore fiducial markers.
 2. The medical device of claim 1, wherein the oneor more fiducial markers are configured to facilitate optical trackingof one or more positions of an internal portion of the elongated shaftinside the body of the patient.
 3. The medical device of claim 2,wherein the one or more fiducial markers are located on an outer surfaceof the elongated shaft near the proximal end.
 4. The medical device ofclaim 3, wherein the one or more fiducial markers are positioned to bedetected by a single stationary camera irrespective of an orientation ofthe medical device relative to the body of the patient.
 5. The medicaldevice of claim 1, wherein the one or more fiducial markers arepositioned to be detected by a single stationary camera irrespective ofan orientation of the medical device relative to the body of thepatient.
 6. The medical device of claim 1, wherein the one or morefiducial markers are located on an outer surface of the elongated shaftnear the proximal end.
 7. The medical device of claim 1, wherein theelongated shaft comprises a cylindrical tube and the one or morefiducial markers are located on a ring around the cylindrical tube. 8.The medical device of claim 1, wherein the one or more fiducial markersinclude one or more markers along a length of the elongated shaft.